Operations on Multimodal Records: Towards a Computational Cognitive Linguistics

The report discusses a cognitive model in which a key concept is Multimodal Record (MMR)--an organized aggregate of transcripts of signals representing all information an Agent continuously acquired for a certain period of time. The MMR consists of a video track, sound track, and a number of tracks containing transcripts of the values of temperature, pressure, etc., as well as transcripts of states of the Agent's internal structure. Three basic operations on MMRs, i.e. multimodal difference (m-), multimodal union (m+) and multimodal intersection (m*), to be performed using neural network has been introduced. Based on the operations one can explain and/or implement a number of psycho-linguistic phenomena. MMR may be considered as a computable form of Image Scheme--the basic concept of Lakoff-Langacker Cognitive Grammar. Hence, the proposed model seems to be a bridge over the gap between the non-computational Cognitive Linguistics and an applied neurocomputing. Moreover, it may be considered as a step towards a unified symbolic-connectionist paradigm.